Skip to content

ScheduledMerges: make the write buffer size configurable #716

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
May 21, 2025

Conversation

jorisdral
Copy link
Collaborator

@jorisdral jorisdral commented May 8, 2025

We'd need to configure the write buffer size in the ScheduledMerges prototype if we want to compare traces for the prototype and real implementation (see #445).

This PR also changes the run size calculations to be somewhat closer to the calculations in the real implementation.

This PR builds on top of #715 to reduce the number of merge conflicts

@jorisdral jorisdral self-assigned this May 8, 2025
@jorisdral jorisdral force-pushed the jdral/restructure-prototypes branch from 3d24f3d to e9f008f Compare May 8, 2025 15:21
@jorisdral jorisdral force-pushed the jdral/prototypes-configurable-writebuffersize branch from 8c8c7ee to ffc0607 Compare May 8, 2025 15:23
@jorisdral jorisdral force-pushed the jdral/restructure-prototypes branch from e9f008f to bb50a83 Compare May 13, 2025 11:37
@jorisdral jorisdral force-pushed the jdral/prototypes-configurable-writebuffersize branch 2 times, most recently from 5244165 to 32aa054 Compare May 13, 2025 11:47
@jorisdral jorisdral marked this pull request as ready for review May 13, 2025 11:50
@jorisdral jorisdral marked this pull request as draft May 14, 2025 14:17
@jorisdral
Copy link
Collaborator Author

I've put the PR back into draft mode because I found a bug

@jorisdral jorisdral force-pushed the jdral/prototypes-configurable-writebuffersize branch 4 times, most recently from 2b67d45 to 44a8197 Compare May 15, 2025 15:08
@jorisdral
Copy link
Collaborator Author

Ready for review again!

@jorisdral jorisdral marked this pull request as ready for review May 15, 2025 15:12
@jorisdral jorisdral force-pushed the jdral/prototypes-configurable-writebuffersize branch from 44a8197 to ad1e471 Compare May 15, 2025 15:46
Base automatically changed from jdral/restructure-prototypes to main May 21, 2025 10:28
Copy link
Collaborator

@dcoutts dcoutts left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable.

@dcoutts dcoutts added this pull request to the merge queue May 21, 2025
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks May 21, 2025
@jorisdral jorisdral force-pushed the jdral/prototypes-configurable-writebuffersize branch from ad1e471 to 18063cb Compare May 21, 2025 11:24
@jorisdral jorisdral enabled auto-merge May 21, 2025 11:24
jorisdral added 3 commits May 21, 2025 15:13
…uffer size

This config value is largely ignored for now, but it is passed to places where
we'll need it in the next few commits.
The new functions can compute the maximum run size for a tiering/levelling run
given a level number, or conversely, can compute the level number of a
tiering/levelling run given the run's size. These alternative calculations are
parameterised by the maximum write buffer size, which is configured using the
`configMaxWriteBufferSize` field of `LSMConfig`.

As a sanity check for the run size calculations, some properties are added to
the prototypes test suite.

These alternative calculations are also closer to the real implementation of the
merge schedule.
And add a test, and immediately start using the alternative calculations in the
prototype.
@jorisdral jorisdral force-pushed the jdral/prototypes-configurable-writebuffersize branch from 18063cb to c0e13ef Compare May 21, 2025 13:14
@jorisdral jorisdral changed the base branch from main to jdral/stylish-haskell-0.15.1.0 May 21, 2025 13:15
@jorisdral jorisdral disabled auto-merge May 21, 2025 13:15
Base automatically changed from jdral/stylish-haskell-0.15.1.0 to main May 21, 2025 13:27
@jorisdral jorisdral added this pull request to the merge queue May 21, 2025
Merged via the queue into main with commit fc417d9 May 21, 2025
28 checks passed
@jorisdral jorisdral deleted the jdral/prototypes-configurable-writebuffersize branch May 21, 2025 14:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants